Automatic Image Annotation Using Adaptive Color Classification

combine multiple cues, such as color, texture, shape, and layout, frequently without user interaction. These include We describe a system which automatically annotates images with a set of prespecified keywords, based on supervised color IBM’s Query-by-Image-Content (QBIC) [10], MIT’s Phoclassification of pixels into N prespecified classes using simple tobook [11], the Trademark and Art Museum applications pixelwise operations. The conditional distribution of the chrofrom ETL [12], Xenomania from the University of Michiminance components of pixels belonging to each class is modgan [13], and Multimedia/VOD testbed from Columbia eled by a two-dimensional Gaussian function, where the mean University [14]. vector and the covariance matrix for each class are estimated Color is a vital part of our everyday experience, and is from appropriate training sets. Then, a succession of binary useful or even necessary for powerful visual processing in hypothesis tests with image-adaptive thresholds has been emthe world around us. Two general approaches have been ployed to decide whether each pixel in a given image belongs employed in the most recent systems for color-based image to one of the predetermined classes. To this effect, a universal annotation: average color and histogram color. Average decision threshold is first selected for each class based on recolor queries are based on the average Munsell color of ceiver operating characteristics (ROC) curves quantifying the an image or objects in the image. A desired color is usually optimum ‘‘true positive’’ vs ‘‘false positive’’ performance on the training set. Then, a new method is introduced for adapting selected from a color palette, and closest matches (within these thresholds to the characteristics of individual input ima Euclidean distance threshold specified by the user) are ages based on histogram cluster analysis. If a particular pixel is returned [10]. Histogram color is based on 3-D color histofound to belong to more than one class, a maximum a posteriori gram of an image or objects in the image. Queries are probability (MAP) rule is employed to resolve the ambiguity. formed by specifying a query histogram. Images (objects) The performance improvement obtained by the proposed adapwith similar color histogram are returned. Clearly, use of tive hypothesis testing approach over using universal decision average color or histogram color within an object requires thresholds is demonstrated by annotating a database of 31 prior segmentation (manual or automatic) of the image. images.  1996 Academic Press, Inc. Several methods have been published for automatic color segmentation, including multihistogram analysis [15–18], clustering [19, 20], and Bayesian [21–25] methods. At